Musical Interface to Audiovisual Corpora of Arbitrary Instruments

We present an instrument for audio-visual performance that allows to recombine sounds from a collection of sampled media through concatenative synthesis. A three-dimensional distribution derived from feature-analysis becomes accessible through a theremin-inspired interface, allowing the player to shift from exploration and intuitive navigation toward embodied performance on a granular level. In our example we illustrate this concept by using the audiovisual recording of an instrumental performance as a source. Our system provides an alternative interface to the musical instrument’s audiovisual corpus: as the instrument’s sound and behavior is accessed in ways that are not possible on the instrument itself, the resulting non-linear playback of the grains generates an instant remix in a cutup aesthetic. The presented instrument is a human-computer interface that employs the structural outcome of machine analysis accessing audiovisual corpora in the context of a musical performance.